I'd consider moving the provider strategy outside of the provider subsystem. That is, have the provider subsystem only handle tracking/enqueuing.

A "strategy" would then be a function like:

type ProviderStrategy = func(*Provider, cid.Cid) error
func ProvideRoots(p *Provider, root cid.Cid) error {
    return p.Provide(root)
}
// (a trivial example)

Rational:

1. We're going to need to choose strategies at runtime. We could also do that by passing them into Provide but, IMO, applying the strategy outside gives us more flexibility.
2. We're probably going to want to record chosen strategies elsewhere (e.g., in pin metadata) so we're going to need to manage this information elsewhere anyways.

Thoughts?

This makes sense. I'm going to address this feedback last so we can discuss it in more detail after I get the other feedback incorporated. Briefly, though, my thoughts are:

Would we still need to allow a "global" providing strategy to be configured? How does that work with the ability to call provider strategies explicitly? Would we need to use an abstraction at all of the provide call sites that chooses the actual strategy to use based on the config?

Would we need to use an abstraction at all of the provide call sites that chooses the actual strategy to use based on the config?

Yes. The issue is that we'll likely want different default strategies for different commands (files, other data, pinning, etc.). For example, a "provide every directory/file root" strategy won't make sense as a default for the ipfs dag or ipfs block commands because those work directly with IPLD blocks (which may not be files).

However, it may make sense to introduce another service for this. Maybe? That is, something with a function like GetStrategyFor(stuff) (Strategy, error) where stuff includes things like:

• Whether or not we're pinning.
• The type of data we're handling (raw blocks, ipfs files, etc.).
• Maybe even the root CID?
• Maybe even the current command?

Really, I'm not sure what the right design is here.

Understood, I'm processing this now.

@Stebalien It sounds like the plan is to move the provide calls to the call-sites and to choose an appropriate providing strategy for each location. However, there is currently the ability to configure a providing strategy, which we've expressed the desire to preserve. I'm trying to figure out exactly what those configurations would do.

I think a reasonable approach is to determine all the places where a "provide all" currently takes place (e.g. ipfs add) and to allow those locations to be configured by the user's provider strategies. This makes the system behave a little different from the way it behaves right now, which I think is unavoidable, but achieves the larger goal of allowing a limited strategy to be used in place where there's the potential to provide too many blocks.

Any thoughts?

Just to clarify, in the locations where a "provide all" does not make sense, we just always use whatever providing strategy we decide makes the most sense in that location. Which I think just means calling Provider.Provide(cid) directly.

I think a reasonable approach is to determine all the places where a "provide all" currently takes place (e.g. ipfs add) and to allow those locations to be configured by the user's provider strategies.

Exactly.

This makes the system behave a little different from the way it behaves right now, which I think is unavoidable

Correct. Really, we can probably deprecate the current reprovider strategy system and write a config migration.

For example, we could (eventually) end up supporting a provider strategy spec like the following:

{
  // Default strategy.
  "default": {
    // all objects.
    "all": {
      "strategy": "root",
      // put it in the queue, announce once, but don't record or reprovide
      // We probably won't support this out of the box...
      "when": "once",
    }
  }
  // For `ipfs add --pin=false`. That is, data _we've_ chosen to add (but not pin).
  "add": {
    "all": {
      "strategy": "root",
    },
    "unixfs": {
      "strategy": "files",
      "when": "once",
    }
  },
  // Either `ipfs pin /ipfs/...` or `ipfs add ...`
  "pin": {
    // fallback.
    // In this case, it only applies ot `ipfs pin /ipld/Qm...`
    "default": {
      "strategy": "all",
    },
    // Applies to things like `ipfs pin /ipfs/Qm...`
    "unixfs": {
      "strategy": "files",
    },
  },
}

(however, the user should be able to override the strategy when adding/pinning data.